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I.  INTRODUCTION 


1.  PURPOSE  OF  THE  TEST  PACKAGE 

The  purpose  of  this  Test  Package  is  to  present  procedures  which 
can  be  employed  by  a  PSRO  to  assess  the  performance  of  its  data 
subcontractor's  processing  system  in  data  entry,  data  editing,  error 
identification,  and  generation  of  the  quarterly  PHDDS  tape.    This  test 
is  designed  to  demonstrate  that  the  system  under  review  meets  the 
standards  outlined  in  PSRO  Transmittal  No.  28,  dated  January  9,  1976, 
and  the  Provisional  Policies  for  PSRO  Data  Routing  and  Processing, 
issued  in  September  1975. 

It  is  envisioned  that  the  materials  and  procedures  within  this 
manual  may  be  used  by  a  PSRO  as  a  "system  acceptance  test"  of  the  data 
processing  subcontractor's  system  for  data  entry,  editing,  error 
reporting  and  corrections,  and  PHDDS  tape  generation.     Such  an 
acceptance  test  might  be  used  when  a  system  first  becomes  operational, 
again  following  any  major  coding  or  programming  revisions,  or  for  a 
system  performance  review  scheduled  at  infrequent  intervals.  Since 
the  PHDDS  represents  only  a  portion  of  PSRO  data  needs,  the  procedures 
described  in  the  manual  can  be  extended  to  allow  PSROs  to  include 
tests  of  locally  designated  or  "optional"  items. 

2.  OBJECTIVES 

The  objectives  of  the  Test  Package  are: 

To  identify  error  conditions,  such  as 
Erroneous  or  invalid  codes 
Omission  of  required  codes 

Sex  specific  diagnostic  or  procedural  categories 

Erroneous  or  illogical  dates 

Erroneous  or   illogical  ranges  of  codes 

To  identify  errors  occurring  during  data  transfer  or  storage 

To  identify  errors  occurring  during  processing. 

The  objectives  will  be  achieved  by  use  of  a  set  of  test  records; 
that  is,  by  the  entering  of  a  number  of  abstracts  coded  to  contain 
selected  errors  in  the  PHDDS  items.     The  test  consists  of  two  major 
phases:  errors  must  be  identified  by  the  edit  process  used  by  the 
subcontractor  on  incoming  data,  and  the  contents  of  the  resulting  PHDDS 
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data  tape  must  be  verified  as  accurate  when  all  identified  errors  have 
been  corrected  and  reformatting  for  the  PHDDS  tape  has  taken  place. 
When  both  tasks  are  accomplished,  the  processor's  system  will  be 
considered  to  have  successfully  passed  the  test. 

Although  the  Test  Package  will  include  only  PHDDS  items,  each 
PSRO  will  have  the  ability  to  add  tests  for  local,  optional  data  items. 
In  addition,  if  the  edits  required  by  a  PSRO  are  more  stringent  than 
those  required  by  Transmittal  No.  28,  changes  may  be  made  to  the  test 
abstracts  in  accordance  with  local  requirements.     Specifically,  when 
the  PSRO  requires  the  processor  to  perform  calculations  from  input 
data  -  such  as  determining  age  from  date  of  birth  -  tests  for  the 
accuracy  of  these  activities  may  be  added. 

It  is  important  to  note  that  it  is  not  appropriate  to  merely 
correct  an  error  identified  by  a  processor's  edit.    It  is  appropriate 
to  correct  the  error  and  eliminate  the  cause  of  such  errors.  For 
example,  if  Zip  Code  was  not  entered  on  a  discharge  abstract  and 
inadvertantly  passed  local  edits,  it  would  identified  by  the  Federal 
Processor  or  BQA  as  an  error.     It  would  be  possible  to  enter  the  Zip 
Code  as  "XXXXX"  and  resubmit  the  record,  which  would  now  pass  the  BQA 
edit.    The  fact  that  the  record  successfully  passed  the  edit  indicates 
a  deficiency  in  the  edit  subroutine  associated  with  the  Zip  Code 
element;  the  identified  error  should  be  corrected.    The  same  logic 
underlies  this  test. 

3.        CONTENTS  AND  USE  OF  THIS  DOCUMENT 

This  document  may  be  considered  as  a  "procedure  manual,"  providing 
a  detailed  description  of  each  step  to  be  followed  during  the  testing. 
Data  processing  terms  have  been  avoided  where  possible  and  every 
attempt  has  been  made  to  write  all  instructions  and  descriptions  in 
non- technical  language. 

The  chapters  in  this  document  are  briefly  described  below: 

Chapter  I,  Introduction,  describes  the  purpose  and  objectives 
of  the  Test  Package  in  order  to  provide  a  common 
understanding  and  point  of  departure  for  all  potential  users 

Chapter  II,  Overview  of  the  Test  Process,  explains  the 
testing  process  —  the  sequence  of  events  from  preparation 
of  the  input  data  through  evaluation  of  the  output 

Chapter  III,  Data  Errors  and  Edit  Specifications,  explains 
the  different  coding  issues,  the  edits  involved  in  the  test, 
the  various  edit  levels  and  how  they  are  applied 

Chapter  IV,  identifies  the  Administrative  Procedures  and 
preparatory  steps  necessary  prior  to  beginning  testing 
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Chapter  V,  First  Runf  explains  the  original  entry  and 
processing  of  data,  expected  outi    b,  and  the  necessary 
cor  rec tions 

Chapter  VI,  Second  Run,  explains  the  entry  and  processing  of 
second  run  data,  expected  output,  and  the  necessary 
corrections  to  be  made 

Chapter  VII,  Third  Run,  explains  the  entry  and  processing  of 
the  third  run  data,  expected  output  output,  and  the  necessary 
cor  rections 

Chapter  VIII,  Final  Run,  details  the  necessary  actions  for 
the  last  run  of  the  test  series 

Chapter  IX,  Test  Evaluation,  provides  the  details  for  the 
evaluation  of  the  data  and  the  test  evaluation  report. 

Use  of  this  manual  will  be  facilitated  by  thorough  understanding 
on  the  part  of  PSRO  staff  of  the  requirements  or  standards  which  the 
processor's  system  must  meet  and  of  the  characteristics  of  PHDDS  data 
collection  and  processing.     Transmittal  No.  28  is  included  as  an 
Appendix  to  the  manual  because  of  its  importance  as  a  reference  for 
purposes  of  PHDDS  processing  and  this  Test  Package. 

Accompanying  this  document  (Volume  I)   are  two  other  documents 
which  complete  the  PSRO  Test  Package.    Volumes  II  and  III  contain  the 
sample  discharge  abstracts  used  in  the  test.    Volume  II  is  coded  using 
the  HICDA-2  classification  system  and  Volume  III  is  coded  in  ICDA-8. 
All  entries  other  than  the  diagnosis  and  procedure  codes  are  identical, 
as  explained  in  greater  detail  in  subsequent  chapters.    These  abstracts 
are  samples  only,  designed  to  present  data  items  coded  to  contain 
specific  error  conditions.    This  information  must  be  recoded  to  the 
abstract  or  data  entry  format  used  by  the  individual  PSRO  before  the 
test  can  be  run. 

While  repeated  reference  is  made  to  "the  PSRO  processor" 
throughout  this  Manual,  it  is  important  to  bear  in  mind  that  the  system 
test  described  can  also  be  applied  to  any  system  collecting  and 
processing  PHDDS  data  prior  to  it  being  forwarded  to  the  area 
processor;   tests  of  "collection  systems"  which  are  components  of  the 
overall  PSRO  data  system  are  highly  desirable. 
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OVERVIEW  OF  THE  TEST  PROCESS 
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II.     OVERVIEW  OF  THE  TEST  PROCESS 


In  order  for  a  PSRO  to  declare  a  processor's  system  fully 
operational,  the  system  should  be  tested  for  acceptable  performance. 
That  is,  data  must  be  entered  into  the  system,  edited,  processea,  and 
reports  generated  and  compared  against  known  standards.     Testing  of 
reports  generated  by  the  processor  for  PSRO  use,  except  for  correction 
of  data  items,  will  not  be  part  of  this  Test  Package;  however,  the  PSRO 
may  want  to  include  such  tests  in  the  acceptance  requirements  for  the 
processor. 

The  data  which  will  comprise  the  test  will  be  coaed  with  known 
errors,  intended  to  represent  errors  which  may  occur  in  the  recoraation 
of  data  on  abstracts.     Knowing  the  error  content  and  correctness  of 
the  test  data  in  advance  serves  as  the  basis  for  assessing  the  integrity 
of  the  processor's  computerized  edit  functions.    Test  results  which 
verify  the  expected  results  pass  the  test;  test  results  which  do  not 
correspond  with  the  expected  results  must  be  examined  and  the  reason (s) 
for  the  discrepancies  determined.     In  this  way,  the  PSRO  and  the 
subcontractor  can  assure  the  logical  accuracy  of  the  data  actually 
forming  the  PSRO's  database  and  being  reported  to  BQA. 

Exhibit  II-l,  on  the  following  page,  provides  a  flow  diaqram  for 
the  test.     The  stages  of  the  test  are  divided  into  major  functional 
areas,  each  one  identifying  what  is  to  be  accomplished  and  how  to 
accomplish  it.     The  major  steps  are: 

Data  Recording,  Collection,  and  Entry 

Data  Editing  and  Correction 

PHDDS  Tape  Generation 

Test  Evaluation. 

Each  step  involves  an  iterative  process  of  checking  actual  results 
against  expected  results  and  determining  the  cause  and  acceptability 
of  any  discrepancies  which  may  appear.     This  Manual  proviaes 
instructions  for  executing  each  step.    In  the  remainder  of  this  chapter, 
the  four  major  steps  will  be  briefly  discussed.     Detailed  directions 
for  their  performance  will  be  provided  in  subsequent  chapters. 
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EXHIBIT  11-1 
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1.        DATA  RECORDING,  COLLECTION,  AND  ENTRY 


The  first  major  step  of  every  PSRO  data  system  covers  the  process 
of  abstracting  medical  history  and  review  data  from  patient  recoras, 
and  forwarding  these  documents  to  the  data  collector/processor  for 
entry  into  the  computer  for  editing.     For  purposes  of  this  test,  test 
abstracts  will  be  coded  and  entered  into  the  computer. 

The  actual  data  entry  technique  employed  will  be  dependent  upon 
the  local  routing  and  processing  system.     The  methods  most  commonly 
used  are  key  punching  from  hard-copy  abstracts,  optical  scanning  of 
hard  copy  abstracts,  key-to-tape  or  key-to-disk  from  hard  copy 
abstracts,  and  on-line  entry  via  a  cathode  ray  tube   (CRT).     In  some 
instances,  data  processors  receive  the  abstracts  already  on  magnetic 
tape  from  hospitals  with  in-house  computer  system  or  from  discharge 
abstract  systems,  so  tape-to-tape  entry  is  used.   Any  mode  is  acceptable 
for  test  purposes. 

2.        DATA  EDITING  AND  CORRECTION 

Once  entered  into  the  processing  system,  the  second  step  is  to 
edit  the  data  for  errors,  omissions,  or  certain  logical  inconsistencies 
in  the  information  recorded.     This  is  the  heart  of  the  Test  Package. 
As  stated  previously,  known  errors  will  be  entered  into  the  edit 
programs;   the  goal  is  to  see  if  the  errors  are  identified  by  the 
computer.     Following  identification  of  each  error,  it  shoula  be 
corrected  using  correction  procedures  as  close  to  normal  as  possible. 
The  corrections  will  then  be  applied  to  the  original  and  further 
editing  and  processing  will  follow. 

The  test  abstracts  are  grouped  into  batches  representing 
discharges  from  several  test  hospitals.     It  is  planned  that  the  data 
entry  and  correction  process  will  involve  several  cycles  before  "clean" 
data  will  be  attained.    As  described  in  detail  in  later  chapters,  three 
runs  and  three  correction  cycles  are  envisioned,  but  additional  runs 
may  be  required  if  the  edit  programs  do  not  identify  all  expected 
errors.     The  run  series  is  designed  to  test  for  the  entry  of  an  error 
during  the  correction  process,  as  well  as  during  original  data  entry. 
Between  each  run,  careful  checking  of  actual  results  as  compared  to 
expected  results  will  be  required. 

Special  procedures  are  required  when  the  data  are  entered  on- 
line through  a  CRT.     In  these  cases,  many  errors  will  be  immeaiately 
identified  by  the  computer  as  the  data  is  keyed.     The  cycles  of  runs 
and  corrections  may  be  adapted  for  use  of  the  Test  Package  on-line, 
as  described  in  Chapter  V. 
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3.        PHDDS  TAPE  GENERATION 


When  all  known  errors  are  identified  by  the  system  and  corrections 
have  been  applied,  a  PHDDS  tape  should  be  generated.    A  tape  dump  (i.e., 
contents  of  the  PHDDS  tape)    will  then  be  studied  to  assure  that  no 
data  transposition,  loss,  or  alteration  has  occurred  in  the  process  of 
moving  the  discharge  data  from  the  PSRO's  data  base  to  the  tape. 

4.        TEST  EVALUATION 

It  is  expected  that  the  edit  correction  process  described  above 
may  require  some  alteration  of  the  processing  system  if  problems  are 
identified  in  the  edit  process.  The  PSRO  will  have  the  ability  to 
assure  that  the  processor's  system  does  inoeed  meet  the  minimum  eoit 
requirements  established  in  Transmittal  No.  28  or  the  more  stringent 
requirements  imposed  by  the  PSRO  on  its  discharge  data.  Evaluation 
of  subcontractor  performance  can  be  made  following  the  test. 

To  summarize  the  steps  of  the  test,  Exhibit  II-2  on  the  following 
page  may  serve  as  a  handy  guide.     It  illustrates  the  iterative  nature 
of  the  test  process  and  the  interrelationships  among  the  various 
stages. 
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EXHIBIT  11-2 
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DATA  ERRORS  AND  EDIT  SPECIFICATIONS 


III.     DATA  ERRORS  AND  EDIT  SPECIFICATIONS 


Since  the  purpose  of  this  Test  Package  is  to  provide  a  means  for 
assuring  that  normal  data  errors  found  in  abstracts  are  detected  by 
the  collector/processor  responsible  for  data  editing,  a  systematic 
representation  of  the  errors  which  may  occur  is  necessary.     In  this 
chapter,  the  actual  errors  and  their  types  are  described. 

1.        DIFFERENCES  BETWEEN  LOCAL  CODING  PROCEDURES  AND  TRANSMITTAL  NO. 
28  REPORTING  SPECIFICATIONS 

In  order  to  record  the  detail  needed  for  local  use,  many  PSROs 
collect  the  data  items  comprising  the  PHDDS  differently  from  the 
required  format  for  reporting  that  information  to  BQA  op  the  quarterly 
PHDDS  tape.     For  example,  sex  may  be  recorded  on  the  PSRO's  abstract 
as  "M,"  "F,"  or  "0"  for  other,  although  for  PHDDS  purposes,  sex  must 
be  reported  as  "1"  for  male  or  "2"  for  female.    Another  commonly  found 
example  is  when  numerous  race  categories  are  included  on  the  PSRO's 
abstract  to  allow  additional  detail  on  minorities  to  be  collected. 
Other  examples  are  also  available. 

In  reporting  this  information  on  the  PHDDS  tape,  these  data  items 
must  be  recoded  so  that  the  data  falls  within  the  required  constraints. 
As  described  above,  "M"  for  sex  must  be  recoded  to  "1";   for  the  race 
element,  all  patients  must  be  recoded  as  either  "1"  for  white,  "2"  for 
black,  or  "3"  for  other.    The  PHDDS  tape  also  is  required  to  have  the 
elements  in  a  certain  order   (date  of  birth,  followed  by  sex,  followed 
by  race)    and  to  have  each  element  be  a  specific  prescribed  length 
(date  of  birth  is  the  first  seven  positions  in  the  record,  followed 
by  sex  with  the  eighth  position,  and  race  in  the  ninth,  etc.)  .  This 
reformatting  and  transformation  of  data  is  thus  an  important  part  of 
PHDDS  tape  production  and  cannot  be  assumed  to  be  performed  correctly 
without  review. 

Additionally,  certain  items  to  be  reported  on  the  PHDDS  tape  may 
not  be  coded  on  the  abstract  but  may,  in  some  cases,  be  generated  by 
the  computer  during  the  preparation  of  the  tape.     In  general,  these 
items  do  not  vary  from  record  to  record,  which  is  to  say  that  they  are 
PSRO-specif ic.     Likely  to  fall  into  this  category  are: 

PSRO  Identifier 

Hospital  Code 

Century  Indicator 

Code  Flag 
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Diagnosis  Flag 


Procedure  Flag. 

This  Package  attempts  to  test  all  possible  errors  of  whatever 
kind  that  may  occur.    When  the  contents  of  an  item  are  computer- 
generated,  the  tests  included  in  the  test  abstracts  may  not  be 
applicable,  although  the  output  results  —  the  PHDDS  tape  —  must  still 
be  verified  for  accuracy.    Local  coding  differences  must  be  accounted 
for  in  running  the  test  as  well  as  in  coding  the  test  abstracts  for 
original  input. 

2.        TYPES  OF  ABSTRACT  ERRORS 

The  errors  which  may  occur   in  coding  abstracts  vary  in  severity 
and  type.    Some  errors  may  simply  involve  miscoding  an  item  or  omitting 
it;  others  involve  the  relationship  between  two  or  more  items,  at  least 
one  of  which  may  be  miscoded.      Other  errors  involve  "key"  data  items 
without  which  further  processing  cannot  occur.     In  the  following 
paragraphs,  the  types  of  errors  which  this  test  package  is  designed 
to  test  are  categorized  and  briffly  described.    Three  "levels"  of 
errors  are  involved. 

(1)     Level  I:     "Fatal"  Errors  in  Key  Items 

Most  PSRO  systems  require  that  certain  data  items  must  be 
properly  recorded  on  the  abstract  for  the  system  to  accept  the 
record  for  processing.    These  "key"  items  are  mandatory  for 
identification  of  the  abstract  or,  because  of  their 
inter  relationship ( s)   with  other  items  in  the  record,  for 
completing  the  edit  of  the  record.     Specifically,  the  type  of 
items  which  are  frequently  designated  as  "key"  are: 

Patient  identification  number* 

Patient  date  of  birth 

Discharge  date 

Admission  date 

Hospital  identification  number. 


( *Note:  "Patient  identification  number"  is  not  reported  on  the 
PHDDS  tape  to  the  Bureau  of  Quality  Assurance.  Proper  completion 
of  the  item,  however,  is  obviously  required). 

Each  of  these  items  must  be  properly  completed  in  accordance  with 
local  requirements.  In  some  cases,  non-completion  of  these  items 
may  cause  editing  or  even  data  entry  to  terminate  before  all 
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items  are  processed.    These  "fatal"  errors  must  be  corrected  ana 
the  entire  record  re-edited  or  re-entered,  as  appropriate. 

(2)  Level  II:     Integrity  of  Individual  PHDDS  Items 

The  remainder  of  the  required  PHDDS  items  have  specifiea 
limits  and  coding  characteristics,  accoraing  to  Transmittal  No. 
28.    Each  of  these  items  must  be  verified  (as  were  the  "key"  items 
comprising  Level  I)    for  such  errors  as: 

Non-allowed  blank  (omission) 

Alphabetic  character (s)    in  numeric  field 

Leading  or  following  blanks 

Code  not  within  allowable  range 

Left  or  right  justified  inappropriately 

Diagnosis  or  procedure  codes,  which  are  not  legal  entries, 
according  to  the  HICDA-2  or  ICDA-8  classification  manuals 

Erroneous  Zip  Code  entries  according  to  U.S.  Postal  Service. 

The  editing  to  be  accomplished  in  this  Level  includes  all 
remaining  PHDDS  information  items  as  well  as  "indicators"  which 
are  required  by  BQA,  such  as  the  diagnosis  flag  which  indicates 
whether  secondary  diagnoses  are  recorded  and  reported  at  the  PSRO 
level  and  whether  they  existed  for  each  indiviaual  discharge 
reviewed.  These  items  may  not  be  able  t'o  be  tested  if  generated 
by  the  computer  when  the  PHDDS  tape  is  being  prepared. 

Not  specifically  included  as  PHDDS  elements  but  items  which  should 
be  routinely  edited  for  completion  and  acceptability  are  the 
following : 

Hospital  numbers 

Physician  numbers   (attending  and  operating) 

Patient  identification  numbers. 

Also  subject  to  review  would  be  batch  number,  abstract  number, 
and  PSRO  number,  when  appropriate  to  the  particular  system.  Of 
course,  optional  items  collected  by  the  PSRO  may  also  be  routinely 
tested  for  similar  errors. 

(3)  Level  III:     Interrelationships  Among  PHDDS  Items 

The  logical  relationships  among  PHDDS  items  are  tested  for 
reasonableness  under  Level  III  tests.     Many  of  the  limits  for 
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these  relationships  are  specified  in  Transmittal  No.  28.  One 
example  is  the  sex-specific  procedures  and  diagnoses  inaicatea 
in  the  Transmittal;   another  might  be  the  fact  that  "Basis  for 
Assignment  of  Initial  Length  of  Stay"  must  be  codea  with  blanks 
if  the  "Admission  Certification  Process"   item  indicates  that 
admission  review  was  not  performed. 

Extremely  sophisticated  and  complex  logic  checks  may  be 
programmed  into  a  computer  to  assist  in  determining  the 
reasonableness  of  the  recorded  information.     Some  processors 
include  not  only  a  sex-specific  check  on  diagnostic  ana  procedure 
codes  but  also  an  "age"  check.    For  example,  a  six-year  old  female 
recorded  as  delivering  a  baby  would  be  noted  as  a  possible  error 
if  age  checks  are  performed;   this  would  not  occur  if  only  a  sex- 
diagnosis  check  was  made. 

Examples  of  errors  to  be  found  in  Level  III  testing  are 
listed  below: 

Sex-specific  diagnosis  and  patient  of  opposite  sex 

Sex-specific  procedure  and  patient  of  opposite  sex 

Date  of  birth... 

After  date  of  admission 

After  date  of  discharge 

Sex  omitted  with  sex-specific  diagnosis  or  procedure 

Date  of  birth/sex  inappropriate  for  expected  source  of 
payment   (Title  V  patients) 

Discharge  date  before  admission  date 

Admission  certification  not  performed  or  denied,  but  days 
recorded  as  having  been  assigned  at  admisstion 

Physician  advisor  reviews  not  noted  when  denial  occurs. 

The  Level  III  type  errors  are  extremely  important  relationships 
to  systematically  examine  during  a  test  of  data  edits,  for  errors 
of  this  kind  can  seriously  undermine  data  integrity. 

3.        EDIT  STANDARDS 

The  edit  standards  to  be  applied  to  each  item  recorded  for  PHDDS 
tape  generation  are  outlined  in  Transmittal  No.  28  (see  Appenaix) .  For 
purposes  of  documenting  clearly  the  kinds  of  errors  which  tnis  Test 
Package  is  designed  to  test,  an  item-by-item  aescription  of  possible 
errors  is  presented  in  Exhibit  III-I  on  the  following  pages.  This 
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exhibit  is  designed  to  serve  as  a  reference  for  preparing  edit 
specifications  and  criteria  to  be  used  during  system  development  and 
during  system  operation;  although  the  list  is  not  exhaustive,  it 
includes  most  error  types  found  in  abstract  preparation. 

Some  processor's  systems  may  utilize  a  "zero  default"  for  a  blank  if 
the  blank  appears  in  the  rightmost  column.  This  means  that,  for  example, 
a  diagnosis  code  of  340.  will  be  recorded  by  the  system  as  340.0.  All 
coding  on  the  PSRO's  normal  abstracts  should  accurately  reflect  the 
codes  in  the  ICDA-8  or  HICDA-2  manuals.  For  either  system,  if  the 
diagnosis  code  is  a  valid  three-digit  code,  the  fourth  position  should 
be  reported,  entered,  and  stored  as  a  blank  code. 

The  recording  of  duplicate  diagnoses  or  procedures  may  not  be 
considered  an  error  if  the  processor's  system  ignores  the  duplicate 
entry.     Likewise,  a  processor's  system  may  not  consider  a  blank 
diagnosis  or  procedure  an  error  (i.e.,  primary,  first,  second,  and  fourth 
secondary  diagnoses  are  entered;   the  third  secondary  is  blank) . 
Therefore,  the  test  for  these  types  of  "errors"  (while  not  specifically 
PHDDS  errors)   may  not  be  able  to  be  performed. 
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EXHIBIT  III-l 
PAGE  1  Of  10 


DESCRIPTION  OF  POSSIBLE  ERROR  CONDITIONS 


DATA  ELEMENT 


ERROR  CONDITION 


2. 


DATE  OF  BIRTH 


A.     Century  Indicator 


Blank 

Number  Other  than  1    or  0 


B.     Year  of  Birth 


Alpha 
Blank 

<51  and  Century  Indicator 
>77  and  Century  Indicator 


1 

0 


C.     Month  of  Birth 


Alpha  other  than  XX 
Numberic  =  00  or  >12 
Blank 

Numberic  and  Blank 
XX  and  Day  filled  in 


D.     Day  of  Birth  Alpha  Other  Than  XX 

Blank 

Month  =02  and  day  >  28  and 

not  Leap  Year  or  >  29 
if  Leap  Year 

Numeric  =  00  or  >  31  or  >  30 
depending  on  month 


3.  SEX 


Blank 
Alpha 
Numeric 


other  than  1  or  2 
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DESCRIPTION  OF  POSSIBLE  ERROR  CONDITIONS 


DATA  ELEMENT 


ERROR  CONDITION 


RACE 


Blank 

Alpha  other  than  X  for  unknown 
Numeric  other  than  1 ,  2 ,  3 


ZIP  CODE 


Blank 

Alpha  other  than  YYYYY  or  XXXXX 
Mixture  of  alpha  and  numeric  digits 
Invalid  code  for  the  first  three  digits 


HOSPITAL  INDICATOR 


A.     PSRO  ID  Number 


Invalid  Number 


B.     Hospital  Code 
Number 


Blank 
Alpha 

Invalid  number 


ADMISSION  DATE  AND  HOUR 


A.     Date  of  Admission 


B.     Year  of  Admission 


Before  date  of  birth  or  date  of  discharge 

Alpha 
Blank 


C.     Month  of  Admission 


Alpha 

Numeric 

Blank 


=  00  or  >  12 


D.     Day  of  Admission 


Alpha 

Numeric  =  00  or  >  31 
or  >30  depending  on  month 

Month  =02  and  Day  >28 
and  not  leap  year  or  >29  if 
leap  year 
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DESCRIPTION  OF  POSSIBLE  ERROR  CONDITIONS 


DATA  ELEMENT 


ERROR  CONDITION 


E.       Hour  of  Admission 


Alpha 
Blank 

Outside  range  of  00-23 


8.        DISCHARGE  DATE 


A.       Discharge  Date 


Before  date  of  admission  or  date 
of  birth 


B.       Year  of  Discharge 


Alpha 
Blank 


C.       Month  of  Discharge 


Alpha 

Numeric 

Blank 


=  00  or  >12 


D.       Day  of  Discharge 


Alpha 
Blank 

Numeric  =  00  or  >31  or  >30 

depending  on  month 
Month  =02  and  Day  >28 

and  not  leap  year  or  >29 

if  leap  year. 


DIAGNOSIS/PROCEDURE  CODE 

FLAG  Blank 

Other  than  1  or  2 
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DESCRIPTION  OF  POSSIBLE  ERROR  CONDITIONS 


DATA  ELEMENT  ERROR  CONDITIONS 

DIAGNOSIS 

A.       Principal  Diagnosis  Blank 

Blank  in  other  than  right 

most  position 
Alpha  in  other  than  left  most 

position 
Not  Compatible  with  SEX  edit 
Invalid  ICDA-8  or  HICDA-2  Code 
"E"  Code 


B.       Diagnosis  Flag*  Blank 

Alpha 

Numeric  =  0  or  >4 

Value  =  1  and  secondary  diagnosis 
present 

Value  =  2  and  secondary  diagnosis 

not  present 
Value  =  3  and  secondary  diagnosis 

present 

Value  =  4  and  secondary  diagnosis 
not  present 


C.       First  Secondary 
Diagnosis 


Blank  but  diagnosis  flag  =2 
Blank  in  other  than  right -most 

position 
Alpha  in  other  than  left  most 

position 
Not  compatible  with  Sex  edit 
Invalid  ICDA-8  or  HICDA-2  Code 
"E"  Code 

Duplicate  diagnosis 


*Note:     The  Diagnosis  Flag  will  vary  from  record  to  record. 
It  will  most  likely  be  computer  generated  and  not 
subject  to  testing. 
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DESCRIPTION  OF  POSSIBLE  ERROR  CONDITIONS 


DATA  ELEMENT  ERROR  CONDITION 

D.       Second  Secondary 

Diagnosis  Blank  in  other  than  right  most 

position 
Alpha  in  other  than  left  most 

position 
Not  compatible  with  Sex  edit 
Invalid  ICDA-8  or  HICDA-2  Code 
"E"  Code 

Second  Secondary  diagnosis  pre- 
sent but  no  first  secondary 
diagnosis 

Duplicate  Diagnosis 


E.       Third  Secondary 
Diagnosis 


Blank  in  other  than  right  most 

position 
Alpha  in  other  than  left  most 

position 
Not  Compatible  with  Sex  edit 
Invalid  ICDA-8  or  HICDA-2  Code 
"E"  Code 

Third  secondary  diagnosis  pre- 
sent but  no  first  and/or 
secondary  diagnosis 


F.       Fourth  Secondary 
Diagnosis 


Blank  in  other  than  right  most 

position 
Alpha  in  other  than  left  most 

position 
Not  Compatible  with  Sex  edit 
Invalid  ICDA-8  or  HICDA-2  Code 
"E"  Code 

Duplicate  Diagnosis 
Fourth  secondary  diagnosis 
present  but  no  first/second/ 
third  secondary  diagnosis 
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DESCRIPTION  OF  POSSIBLE  ERROR  CONDITIONS 


DATA  ELEMENT 


12.  PROCEDURES 


A.  Principal 
Procedure 


ERROR  CONDITION 


Invalid  ICDA-8  or  HICDA-2  Code 
Not  compatible  with  Sex  edit. 


B.       Date  of  Principal 
Procedure  Year 


Month 


Day 


Date 


Alpha 

Blank,  but.  principal  procedure 
present  >77 

Alpha 

Blank,  but  principal  procedure 

present 
Numeric  =  00  or>  12 
Numeric  and  blank 

Alpha 

Blank,  but  principal  procedure 
present 

Month  =  02  and  Day  >2  8  and  not 
leap  year  or  >29  if  leap  year 

Numeric  =  00  or     >  31,  or  >  30 
depending  on  month. 

Not  between  Date  of  Admission 

and  Date  of  Discharge 
If  columns  59-61  filled  is 

62-67  blank 


C.       Procedure  Flag* 


rNote 


Alpha 
Numeric  =  0  or  >4 
Principal  procedure  blank  but 

this  field  coded 
This  field  =1  but  secondary  pro- 
cedure present 
This  field  =2  and  no  secondary 

procedure  present 
This  field  =3  but  secondary  pro- 
cedure present 
This  field  =4  and  no  secondary 
procedure  present 

The  Procedure  Flag  will  vary  from  record  to  record. 
It  will  most  likely  be  computer  generated  and  not 
subject  to  testing. 
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DESCRIPTION  OF  POSSIBLE  ERROR  CONDITIONS 


DATA  ELEMENT 


ERROR  CONDITION 


D.       First  Secondary 
Procedure 


E.       Date  of  First 
Secondary 
Procedure 


Invalid  ICDA-8  or  HICDA-2  code 
Not  compatible  with  Sex  edit 
This  field  present  but  no  pri- 
mary procedure 


Year 


Alpha 

Blank,  but  first  secondary  pro- 
cedure present 
>77 


Month 


Alpha 

Blank,  but  First  Secondary  Pro- 
cedure present 
Numeric  =  00  or  >12 
Numeric  and  Blank 


Day 


Date 


F.       Second  Secondary 
Procedure 


Alpha 

Blank,  but  First  Secondary  Pro- 
cedure Present 

Month  =.02  and  Day  >28  and  not 
leap  year  or  >29  if  leap  year 

Numeric  =  00  or  >31  or  >30 
depending  on  month. 

Date  present  but  no  first 

secondary  procedure 
Not  Between  Date  of  Admission 

and  Date  of  Discharge 

Invalid  ICDA-8  or  HICDA-2  Code 
Not  compatible  with  Sex  edit. 
This  field  present  but  no  pri- 
mary and/or  first  secondary 
procedure 
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DESCRIPTION  OF  POSSIBLE  ERROR  CONDITIONS 


DATA  ELEMENT 


ERROR  CONDITION 


G.  Date  of  Second 
Secondary  Pro- 
cedure 


Year 


Alpha 

Blank,  but  Second  Secondary  Pro- 
cedure Present 
>77 


Month 


Alpha 

Blank,  but  Second  Secondary 

Procedure  Present 
Numeric  -  00  or  >12 
Numeric  and  Blank 


Day 


Date 


H.       Third  Secondary 
Procedure 


Alpha 

Blank,  but  Second  Secondary 
Procedure  Present 

Month  =  02  and  Day  >28  and  not 
leap  year  or     >29  if  leap  year 

Numeric:     00  or  >31  or  >30  de- 
pending on  month. 

Date  present  but  no  second  secon- 
dary procedure 

Not  Between  Date  of  Admission 
and  Date  of  Discharge 

Invalid  ICDA-8  or  HICDA-2  Code 
Not  compatible  with  Sex  edit 
This  field  present  but  no  prin- 
cipal and/or  first  secondary 
and/or  second  secondary 
procedure  present 


EXHIBIT  III-l 
Page  9  of  10 


DESCRIPTION  OF  POSSIBLE  ERROR  CONDITIONS 


DATA  ELEMENT 


ERROR  CONDITION 


I.       Date  of  Third 
Secondary 
Procedure 

Year 


Alpha 

Blank,  but  Third  Secondary 

Procedure  present 
>77 


Month 


Alpha 

Blank,  but  Third  Secondary 

Procedure  present 
Numeric  =  00  or  >12 
Numeric  and  Blank 


Day 


Date 


Alpha 

Blank,  but  Third  Secondary 

Procedure  present 
Month  =  02  and  Day  >28  and 

not  leap  year  or  >29  if  leap 

year . 

Numeric  =  00  or  >31  or  >30 
depending  on  month. 

Date  present  but  no  third  secon- 
dary procedure 

Not  Between  Date  of  Admission 
and  Date  of  Discharge. 


13 


PATIENT  DISPOSITION 


Blank 
Alpha 

Not  left  justified 
Numeric  =00  or  >08 


14 .      EXPECTED  PRINCIPAL 
SOURCE  OF  PAYMENT 


Blank 
Alpha 

Not  left  justified 
Numeric  =  00  or  >10 

If  Title  V  payment  and  female 

is  over  age  28 
If  Title  V  payment  and  male  is 

over  21 


15.      NUMBER  OF  DAYS 
CERTIFIED  AT 
ADMISSION 


Blank 
Alpha 
Mixture 
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DESCRIPTION  OF  POSSIBLE  ERROR  CONDITIONS 


DATA  ELEMENT 


ERROR  CONDITION 


16.     ADMISSION  CERTIFICATION 
PROCESS 


Blank 
Alpha 
Numeric  >4 

If  this  field  =3,  Error  if 
Following  Codes  Not  Present: 
Days  assigned  at  admission 
Admission  certification  level 
review  =2 

Reviews  to  physician  advisor ±  0 
If  this  field  =4,  error  if 

Following  Codes  Not  Present: 
Days  assigned  at  admission 
Basis  for  initial  05  =  # 
Admission  certification  level 
of  review  =  # 


17.  BASIS  FOR  ASSIGNMENT 
OF  INITIAL  LENGTH  OF 
STAY 

18.  ADMISSION  CERTIFICATION 
LEVEL  OF  REVIEW 


19.     TOTAL  NUMBER  OF  DAYS 
CERTIFIED 


20.  TOTAL  NUMBER  OF  REVIEWS 
REFERRED  TO  PHYSICIAN 
ADVISOR 

21.  TOTAL  NUMBER  OF  EXTEN- 
SIONS APPROVED 

22.  EXTENSION  DENIAL 


Numeric  =  0  or  >2 
Alpha 


Alpha 

If  column  102  =  4,  error  if  not 

#  in  Column  104. 
Numeric  =  or  >  2 

Blank 
Alpha 

Not  left  justified 
If  this  field  =000,  then  error 
if  not: 

Days  assigned  at  admission  =00 
Reviews  to  physician  advisor  >0 


Blank 
Alpha 

Blank 
Alpha 

Blank 
Alpha 
>1 

If  field  =1,  error  if  reviews  to 
physician  advise:: 
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IV.     ADMINISTRATIVE  PROCEDURES 


The  administrative  steps  necessary  to  conduct  the  test  are 
described  in  this  chapter.    The  following  subsections  explain  the  step 
by  step  actions  to  be  taken  in  preparation  for  conducting  the  test. 
The  resources  and  understandings  necessary  for  the  data  processing 
flow  and  interaction  of  the  data  are  discussed.    Details  for  conducting 
the  test  are  provided  in  Chapter  V. 

1.  TEST  ENVIRONMENT 

The  data  processing  system  is  maintained  by  the  processor  and 
consists  of  the  processor's  computer  system,  the  data,  the  programs, 
and  related  files  and  data  bases.    The  data  entry  medium  of  the  system 
test  will  be  determined  by  the  type  of  data  entry  procedures  routinely 
utilized  by  the  processor.    For  example,  if  punched  cards  are  utilized 
by  the  processor,  then  the  data  entry  medium  for  the  test  will  be 
punched  cards. 

The  PSRO  should  be  familiar  with  the  processor's  equipment,  its 
general  capability,  and  location,  so  that  the  test  can  be  conducted  as 
efficiently  as  possible  and  so  that  the  PSRO  can  have  confidence  that 
the  test  has  been  conducted  in  a  way  which  tests  the  normal  operating, 
system.    Availability  of  key  processor  personnel  or  liaison  may  be 
very  helpful  during  the  conduct  of  the  test. 

2.  TEST  COORDINATION 

The  testing  of  the  system  will  require  continuing  coordination 
between  the  processor  and  the  PSRO.    It  may  also  require  hospital  data 
collection  personnel  as  well  as  messenger  or  delivery  service.  The 
PSRO  is  responsible  for  proper  coordination  of  the  test  procedures 
and  personnel.    The  following  items  must  be  considered: 

Test  Schedule 

Computer  time  allocation  for: 
Data  entry 

Editing  and  correcting 

Report  and  tape  generation 

Coordination  of  test  schedule  among  all  participants, 
including 

PSRO  staff 
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Data  processor's  staff 
Data  collector,  if  appropriate 
Staff  allocation  for 

Preparation  of  test  abstracts 
Error  correciton  and  data  analysis 
Documentation  of  results 

Programming  verification  or  change,  if  needed 

Cost  identification  for 

Discharge  abstracts 

Computer  time 

Report  generation. 

It  is  important  that  all  PSRO  and  processor  personnel  involved 
in  the  test  be  aware  of  its  objectives,  what  each  individual  is  expected 
to  do,  and  of  expected  results. 

3.        TEST  MATERIALS  AND  REQUIRED  RESOURCES 

Prior  to  the  test,  certain  materials  must  be  prepared  or  resources 
allocated.     Included  as  required  are: 

From  the  processor 

Details  of  the  edit  and  correction  programs  routinely 
employed  in  processing  the  PSRO's  data 

Test  or  fake  files  for  storing  the  test  data  while  the 
test  is  being  conducted 

Computer  time 

Staff  time  for  data  entry 

Staff  time  for  test  conduct  and  possible  program 
corrections. 

From  the  PSRO 

Space  for  test  materials 

Staff  time  for  preparing  test  abstracts 
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Staff  time  for   test  conduct  and  data  analysis. 

Documentation  for   the  test 

Test  discharge  abstracts  coded  to  contain  specific 
errors 

Test  or  fake  hospital  numbers,  physician  numbers,  and 
patient  numbers 

Documented  expected  results  for  comparison  and 
verification  of  output  from  each  run 

Test  output  (i.e.,  error  listings,  rejected  abstracts, 
dump  of  PHDDS  tape)    for  comparison  expected  results. 

A  key  part  of  preparation  for  the  test  involves  obtaining  and 
coding  the  test  abstracts.    Each  must  be  identifiable  as  a  test  abstract 
to  insure  the  integrity  of  the  PSRO's  data  base.     However,  each  must 
be  treated  as  a  "real"  abstract  for  processing  purposes  during  the 
test.     Coding  of  the  test  abstracts  may  be  expected  to  require  up  to 
40  hours  of  effort  depending  upon  the  local  options  to  be  included 
and  the  complexity  of  the  PSRO's  abstract  or  data  collection  format. 

If  only  the  PHDDS  is  to  be  tested,  the  data  from  the  test  abstracts 
provided  in  this  manual  can  be  transferred  to  the  selected  data  entry 
mode  used  by  the  PSRO  for  the  test,  making  any  necessary  changes  for 
codes  or  ranges  of  values  for  items  are  made.     If  tests  on  local  or 
optional  items  are  to  be  included,  specific  errors  should  be  identified 
and  added  to  select  dummy  abstracts,  and  expected  results  and 
correction  procedures  documented. 

An  additional  important  step  will  be  the  designation  of  fake 
identifiers  for  test  purposes  —  for  the  hospital  number,  patient 
identification  number,  and  physician/surgeon  numbers  recorded.  These 
should  all  be  clearly  identifiable  as  test  numbers,  but  should  also 
be  coded  with  periodic  errors  to  assure  that  proper  editing  is  done 
on  these  fields.    Designation  of  test  identifiers  must  be  coordinated 
with  the  processor. 

4.        TEST  EVALUATION 

The  purpose  of  the  test  is  to  assure  that  data  processing 
procedures  and  error  processing  and  correction  procedures  are 
effective.    The  test  evaluation  procedure  consists  of  review  and 
analysis  of  the  iterative  process  which  consititue  the  operation.  The 
analysis  is  directed  toward  such  areas  as: 

Required  edits  performed 

The  ability  of  the  edit  programs  to  detect,  and  make  proper 
notification  of,  erroneous  input  data 
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Transposition  or  loss  of  data  after  editing. 

At  the  completion  of  tes>  J,  a  brief  report  should  be  produced.  The 

purpose  of  this  report  is: 

To  document  the  results  of  the  test 

To  provide  documentation  for  required  deficiency  correction 
To  provide  documentation  for  user  acceptance. 


IV-4 


) 


V.     FIRST  RUN 


V.     FIRST  RUN 


The  tasks  to  be  performed  for  the  first  run  of  the  Test  Package 
are  more  elaborate  than  for  any  subsequent  run  because  consiaer able 
preparation  is  required  and  because  the  largest  number  of  test 
abstracts  are  involved  in  this  run.     Subsequent  runs  are  for  making 
corrections  to  the  coded  errors  in  order  to  generate  a  perfect  PHDDS 
tape  at  the  end  of  the  last  run. 


The  tasks 

in 

the  first  run  are  the  following: 

TASK 

1  - 

PREPARE  TEST  ABSTRACTS   FOR  DATA  ENTRY 

TASK 

2  - 

PREPARE  SELECTED  FILES  FOR  TEST  DATA 

TASK 

3  - 

ENTER  THE  TEST  ABSTRACTS 

TASK 

4  - 

RUN  TEST  ABSTRACTS  THROUGH  EDIT  PROGRAM 

TASK 

5  - 

COMPARE  OUTPUT  ERROR  MESSAGES  WITH  EXPECTED 

This  task  plan  assumes  that  the  data  processing  subcontractor  has  been 
contacted  by  the  PSRO  regarding  the  nature,  requirements,  and  import 
of  the  test. 

Each  of  the  tasks  will  be  described  in  detail  in  the  following 
sections.  Sample  abstracts  (coded  with  specific  errors)  and  correction 
sheets  to  be  used  with  the  abstracts  are  included  in  separate  volumes 
of  this  manual.  The  abstracts  are  provided  in  two  separate  packages, 
one  for  HICDA-2  coding  and  a  second  for  ICDA-8  coding.  The  abstracts 
are  identical  except  for  the  coding  classification  used. 

TASK  1.        PREPARE  TEST  ABSTRACTS  FOR  DATA  ENTRY 

The  sample  abstracts  presented  in  the  subsequent  volumes  of  this 
Test  Package  are  coded  to  contain  specific  errors  in  the  specific 
sequences.     As  the  first  step  of  the  test,  the  PSRO  must  prepare  data 
input  in  accordance  with  normal  data  preparation  procedures.  The 
following  steps  describe  the  process. 

(1)     Identify  Local  Data  Elements  To  Be  Included  In  The  Test 

The  sample  abstracts  include  all  PHDDS  items   (see  Chapter 
II)    as  described  in  Transmittal  No.  28.     Also  included  on  the 
abstracts  but  not  coded  are  spaces  for: 

PSRO  number 
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Hospital  number 


Attending  physician  identifier 

Operating  physician  identifier 

Admission  review  and  continued  stay  review. 

"Patient  identifier/1  although  not  a  PHDDS  item,  is  coded  to 
facilitate  abstract  identification. 

The  PSRO  planning  to  use  the  Test  Package  must  compare  the 
sample  abstract  to  the  data  entry  form(s)    it  employs.     In  so 
doing,  it  may  determine  whether  it  wishes  to  add  to  the  test  those 
items  or  data  elements  routinely  collected  on  discharges  at  local 
PSRO  option.     It  is  expected  that  the  PSRO's  subcontractor  will 
have  been  provided  edit  specifications  for  these  items  in  addition 
to  the  PHDDS  items,  and  that  a  test  of  these  edits  will  be  helpful 
to  the  PSRO.     In  addition,  if  the  PSRO's  processor  routinely 
performs  calculations  from  the  PHDDS  data   (e.g.,  length  of  stay, 
age)  ,  tests  of  the  accuracy  of  these  calculations  should  be 
planned. 

The  PSRO  should  systematically  determine  the  types  of  errors 
to  be  included  in  the  test  for  these  locally  reportable  items, 
and  should  prepare  test  data  to  be  added  to  selected  records  of 
the  test  abstracts.     An  example  of  how  to  do  this  is  presented 
in  the  Exhibit  on  the  following  page. 

In  some  cases,  the  errors  are  related  to  other  items  and 
more  detail  is  required  for  the  coding  of  errors.     For  example, 
physician  identifiers  may  be  required  to  be  filled,  be  numeric, 
and  be  "legal"  in  the  sense  of  being  listed  by  the  PSRO  as  an 
acceptable  number  for  a  physician  member.    In  this  case,  the  test 
data  should  also  include  a  number  which  is  not  on  the  test  (i.e., 
a  "real"  physician  number)    to  assure  that  the  edit  is  properly 
performed. 

For  items  which  must  be  recorded  for  local  purposes  or  which 
are  otherwise  listed  as  an  error  (e.g.,  admitting  diagnosis) ,  codes 
which  would  never  fail  an  edit  should  be  selected  for  insertion 
on  those  abstracts  which  are  not  testing  the  particular  item.  In 
the  case  of  admitting  diagnosis,  a  code  of  519.2  "Other  Diseases 
of  Lung"  would  be  a  likely  code  to  select. 

Specific  precautions  must  be  taken  in  selecting  local  test 
codes  if  there  is  an  interactive  edit  between  the  local  item  and 
a  PHDDS  element.     Sets  of  these  edits  should  be  confined  to  test 
records  where  either  the  PHDDS  item  is  coded  correctly,  or  is 
specifically  altered  to  test  the  particular  local  edit. 
Otherwise,  tracking  the  case  through  the  test  services  becomes 
burdensome  by  having  to  track  both  sets  of  corrections. 
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EXHIBIT  V-1 


TECHNIQUE  FOR  PREPARING  TEST  DATA 
FOR  NON-REPORTABLE  ITEMS 


ITEM: 

REQUIREMENTS: 
POSSIBLE  ERRORS: 


ERRORS  TO  BE  COOED 


PATIENT'S  SOCIAL  SECURITY  NUMBER 
REQUIRED  FIELD 

MUST  BE  NUMERIC,  9  CHARACTERS 

IF  UNKNOWN,  MUST  BE  XXX  -  XX  -  XXXX 


® 


BLANK  ALTOGETHER 
NON-NUMERIC,  EXCEPT  X's 
ALPHA-FILLED,  EXCEPT  X's 
PARTIALLY  BLANK 
ZERO-FILLED 


ERROR 
TYPE 

ERROR 

CODED  ON 

Batch  No. 

Abstract 
No. 

© 

9 
® 
® 
® 
® 

01 
03 
04 
02 
02 
05 

06 
12 
20 
02 
05 
06 

1 

2 

3  |  —  |  4  |  5  j  —  1 6 

7 

s 

A 

A 

B 

c  |  -|d|  e  ]  -  |f 

G 

H 

I 

Z 

Z 

z  |  -| z  |  z  |  -  |z 

Z 

Z 

z 

6 

7 

2  |  -  |  1  |  3  |  -  | 

0 

0 

0  |  —  |  0  j  0  |  —  [  0 

0 

0 

0 

good 
good 

03 
01 

25 
10 

2 

3 

1  I  -|6  I  4  I  -  [3 

4 

X 

X 

X  J  —  |  X  j  X  j  —  |  X 

X 

X 

X 

(2)     Select  the  ICDA-8  Or  HICDA-2  Abstract  Set 


Each  set  of  abstracts  is  identical  except  for  the  coding 
structure  used.     Each  set  of  abstracts  contains  5  batches  of 
abstracts  with  25,  26,  or  29  abstracts  per  batch  adding  to  a  total 
of  130  discharges  in  the  package.     The  PSRO  should  select  the 
package  for  the  coding  structre  it  uses  to  report  data  to  BQA. 

However,  if  the  PSRO  has  hospitals  reporting  in  both  cooe 
types,  both  packages  may  be  needed  in  certain  circumstances.-  If 
two  or  more  data  collectors  are  used  —  that  is,  if  the  data  are 
edited  by  more  than  one  subcontractor  and  converted  to  one  coding 
structure  by  the  processor  —  all  collectors  should  be  tested. 
If  data  are  received  and  converted  by  a  collector/processor  before 
they  are  edited,  the  test  package  for  the  final  coding  schema  is 
the  only  one  that  need  be  used. 

The  organization  of  these  test  abstracts  should  be  noted. 
Apart  from  the  batch  and  abstract  number,  the  only  item  carried 
through  on  all  runs  is  the  patient  identification  number.  The 
other  entries  on  Run  2  and  Run  3  abstracts  are  corrections  to 
preceding  runs.     There  are  two  implications  to  this: 

If  the  system  needs  additional  items  (such  as  date  of  birth, 
discharge  date,  etc.)    to  locate  the  record  on  a  subsequent 
run,  these  additional  items  must  be  added  to  Run  2  and  Run 
3  abstracts. 

As  an  extension  of  the  above,  in  the  event  the  system  requires 
reentry  of  a  complete  abstract  to  make  corrections,  all 
correct  items  of  earlier  runs  must  be  carried  forward. 

( 3 )     Reorganize  Test  Abstracts  for  Entry,  If  Interactive  Editing 
Occurs 

The  test  abstracts  presented  in  the  attached  volumes  are 
organized  for  a  test  procedure  which  includes  three  run-ana- 
correction  cycles  before  the  PHDDS  tape  is  generated  from  the 
"clean"  data.     A  batch  entry  mode  is  most  likely  to  follow  this 
progression,  with  the  first  run  being  followed  by  correction  runs 
until  clean  records  are  attained.     In  cases  where  the  data  are 
entered  into  the  computer  via  an  interactive  terminal  which  edits 
the  data  during  entry,  some  modification  of  the  test  procedure 
will  be  necessary. 

The  required  modifications  will  mostly  be  in  terms  of 
organization  and  documentation  as  described  below.     Exhibit  V-2, 
on  the  following  page,  illustrates  the  different  approach 

required. 

The  test  abstracts  in  the  attached  volumes  are  organized  so 
that  the  correction  for  abstract  03  in  batch  01,  for  example, 
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ORGANIZATION  AND  DOCUMENTATION  OF 
ABSTRACTS  AND  RESULTS  FOR  TEST  PACKAGE 
IMPLEMENTATION 


EXHIBIT  v- 


HARD  COPY  ABSTRACTS  ENTERED  IN  BATCH  MODE 
(As  organized  in  Volumes  II  and  III) 


BATCH  01 


BATCH  01 


No  2 


BATCH  01 
No  1 


ERROR  LISTING 
BATCH  01 
No  1 
No  3 


ERROR  LISTING 
BATCH  01 
No  3 


ABSTRACTS  FOR  ERROR  LISTING  CORRECTION  ERROR  LISTING 

RUN  1  FROM  ABSTRACTS  FROM 

RUN  1  FOR  RUN  2 

RUN  2 


ERROR  LISTING 

BATCH  01 

BATCH  01 

 H 

No  3 

NO  ERRORS 

CORRECTION 

ERROR  LISTING 

ABSTRACT 

FROM 

FOR 

RUN  3 

RUN  3 

RECORO 1 

RECORO  2 

ETC. 

TAPE  DUMP 
FOR 

RECORO 
VERIFICATION 


ON-LINE  ENTRY  AND  (PARTIAL)  EDITING  INTERACTIVELY 


ALL  ABSTRACTS  (ORIGINAL  AND  CORRECTIONS) 
PERTAINING  TO  AN  INDIVIDUAL  DISCHARGE 


ONLINE  ENTRY  AND  EDITING 
•  Documentation  of  morn 
identified  and  entered 
Oil-Line 


SRROR  LISTING 
BATCH  01 
No  3 

BATCH  01 
No  3 

— H 

RECORD  1 

RECORD  2 

h— ~ 

ETC. 

ERROR  LISTING  FOR 
MORE  SOPHISTICATED 
ERRORS  (LEVEL  III) 


CORRECTION 
ABSTRACT 


GENERATE 
TAPE 


TAPE  DUMP 
FOR  RECORO 
VERIFICATION 


are  found  in  the  "Run  2"  section.     This  is  to  say  that  in 
Run  1,  abstract  03  of  batch  01  is  expected  to  have  certain 
specific  errors  noted  which  should  be  corrected  as  indicated 
on  the  correction  abstract  during  Run  2.     Also,  the  test 

abstracts  have  been  deliberately  printed  on  paper  to  fit  { 
three  ring  binders.    Thus,  for  interactive,  on-line  terminal 
data  entry,  the  test  abstracts  should  be  reorganized  in  the 
following  manner  for  ease  of  data  entry: 

Recollate  the  abstracts;  Run  1,  Batch  01,  abstract 
number  001 (see  upper  right-hand  portion  of  test 
abstract)    is  first 

Next,  turn  to  Run  2,  extract  the  Batch  01,  abstract 
number  001  and  insert  immediately  behind  the  Run  1 
abstract 

There  is  no  Run  3  abstract 

Do  the  same  for  abstract  numbers  002-006 

When  abstract  number  007  is  reached,  there  will  be  a 
Run  3  abstract  to  be  added  after  the  Run  2  abstract. 

Continue  reorganizing  the  rest  of  the  abstracts  in  the 
same  manner,  making  sure  that  the  order  is  Run  1,  Run 
2,  and  Run  3    (as  appropriate)    for  each  batch  and 
abstract  number.     See  the  bottom  half  of  Exhibit  V-2, 
on  the  following  page,  for  a  pictoral  display  of  the 
reorganized  abstracts. 

The  second  area  of  concern  is  the  documentation  of  test 
results.     In  the  batch  mode,  the  PSRO  will  receive  an  error 
listing  from  the  processor  which  will  serve  to  clearly 
indicate  what  errors  were  identified  by  the  edit  program. 
Results  may  be  checked  against  expected  results  in  this  case, 
and  the  error  listing  documents  these  results.     For  an 
interactive  system,  however,  the  PSRO  may  not  receive  an 
error  listing  for  each  error  and  will  therefore  find  it 
harder  to  document  those  errors  which  are  or  are  not 
identified  by  the  system.     In  this  case,  a  device  for 
documenting  the  activity  of  the  terminal,  the  changes  entered 
by  the  terminal  operator,  and  the  results  must  be  developed. 
One  approach  is  to  have  a  PSRO  staff  person  document  the 
errors  as  they  are  detected  and  provide  the  operator  with 
the  appropriate,  correction  information  or  abstracts.  A 
column  for  checking  off  each  corrective  action  is  included 
in  the  sheets  following  the  narrative  in  this  Chapter. 

In  the  event  the  processing  system,  whether  interactive  or 
batch,  does  not  update  or  change  a  data  item  but  instead  requires 
the  record  to  be  deleted  and  a  complete,  corrected  record  to  be 
added,  a  second,  completely  filled-in  record  must  be  prepared. 
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These  adjustments  require  some  planning  on  the  PSRO's  part  but 
are  neither  complex  nor  difficult  to  accomplish.     They  do  not 
alter  the  utility  of  the  Test  Package  to  assess  the  thoroughness 
of  the  subcontractor's  edit  process. 

(4)     Code  Test  Abstracts 

The  sample  abstracts  presented  in  the  attached  volumes  must 
be  coded  into  the  data  format  routinely  used  by  the  PSRO  tor 
entering  its  data  into  the  edit  program  of  the  subcontractor (s) . 
In  many  cases  this  will  mean  that  these  130  abstracts  must  be 
coded  onto  the  hard  copy  abstract  usea  throughout  the  PSRO  area 
for  recording  discharge  data;   in  other  cases,  where  a  discharge 
abstracting  system  is  the  collector  and  editor,  the  abstracts  of 
that  system  must  be  used.     If  a  hospital  collects  and  edits  its 
own  data  using  an  internal  computer  system,  its  format  may  be 
used.     The  exact  process  will  be  determined  by  the  routing  and 
processing  at  the  PSRO;   the  objective  is  to  get  the  coded 
abstracts  into  a  format  such  that  they  can  be  run  through  the 
edit  program(s).     As  previously  mentioned,  the  abstracts  must  be 
coded  according  to  local  conventions,  possibly  involving  some 
translation  to  attain  the  desired  results. 

During  the  abstract  preparation  process,  locally-collected 
items  will  be  coded  on  the  abstracts  in  accordance  with  the 
decisions  made  under  step  1  of  this  task.    The  personnel  involved 
in  the  coding  of  the  test  abstracts  should  be  thoroughly  familiar 
with  the  system  and  associated  coding  procedures  in  order  to 
minimize  the  introduction  of  unintended  errors. 

TASK  2.        PREPARE  SELECT  FILES  FOR  TEST  DATA 

It  is  anticipated  that  preparations  will  be  required  with  the 
processor  to  assure  that  the  test  data  do  not  become  part  of  the  PSRO's 
data  base  and  that  certain  identifiers  acceptable  to  the  system  do 
not  represent  real  individuals. 

The  processor's  tables  of  physician  numbers,  surgeon  numbers,  and 
hospital  numbers  may  require  the  addition  of  one  or  more  entries  to 
assure  that  the  test  data  pass  the  edits  but  are  not  confused  with 
real  individuals  or  organizations.    Although  surgeon  and  physician 
identifiers  are  not  part  of  the  PHDDS  for  reporting  to  BQA,  they  are 
collected  routinely  by  all  PSROs  for  review  purposes.  Review 
coordinator  identifier  may  be  another  such  number.    In  computer  tests, 
fake  identifiers  are  often  constructed  to  be  obviously  unlike  the  real 
ones  so  as  to  avoid  confusion.    For  example,  hospital  numbers  might  be 
all  9's,  8's,  or  7's,  etc.     In  Exhibit  V-3,  a  sample  form  for  noting 
these  necessary  identifiers  is  presented. 

Patient  identifiers  are  also  not  part  of  the  PHDDS  but  are  almost 
universally  collected  by  PSROs  and  may  be  helpful  in  identifying  test 
records  when  batch  and  abstract  numbers  are  only  used  for  batch  control 
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EXHIBIT  V-3 


DUMMY  IDENTIFIER  FORM 


HOSPITAL 
NUMBER 

ATTENDING 
PHYSICIAN 

OPERATING 
PHYSICIAN 

NOTE:  THESE  ENTRIES  MUST  NOT  BE  THE  SAME  AS  LEGITIMATE 
IDENTIFIERS  USED  BY  A  PSRO. 


purposes  and  do  not  become  part  of  the  recora.    Again,  very  distinctive 
but  sequential  patient  identifiers  may  be  useful:     the  sample  test 
abstracts,   for  example,  are  y  character  fields  coaed  from  111111001 
through  111111130.     This  or  some  other  distinctive  technique  may  be 
used  by  the  PSRO. 

For  other  items,  local  usage  may  allow  more  categories  of  response 
than  Transmittal  2  8  or  the  PHDDS  specify.     For  example  according  to 
Transmittal  28,  Race  may  be  recorded  in  3  ways   (l=white,  2=black, 
3=other) ,  but  a  PSRO  may  have  other  codes  such  as  "Native  American", 
"Spanish",  "Oriental",  etc.     In  such  cases,  test  data  may  be  modified 
to  test  for  acceptance  of  these  codes  during  the  edit,  but  the  final 
product  must  adhere  to  PHDDS  format  for  tape  generation. 

In  addition  to  these  adaptations,  the  processor  must  arrange  to 
treat  the  test  data  as  separate  data  from  the  PSRO's  regular  data. 
Separate  files,  tapes,  or  data  base  structures  may  be  required  to  assure 
that  the  test  data  are  segregated.  Since  the  abstracts  were  not  coded 
to  reflect  real  episodes  of  care  but  rather  to  test  error  conditions, 
inclusion  of  these  data  in  the  PSRO's  files  might  strangely  affect  the 
PSRO's  data  base  for  the  time  period. 

TASK  3.        ENTER  THE  TEST  ABSTRACTS 

After  the  abstracts  have  been  prepared  and  the  necessary  file 
entries  have  been  made  for  identifiers,  the  abstracts  can  be  directly 
entered  into  the  computer,  either  in  batch  moae  or  interactively.  In 
the  former  case,  this  step  merely  involves  batching  the  abstracts, 
sending  them  off  to  the  processor,  and  waiting  for  the  error  listing. 
In  the  case  of  the  interactive  terminal,  some  editing  will  be  performed 
during  data  entry  so  specific  procedures  developed  to  follow  and 
document  this  process. 

Data  entry  in  the  batch  mode  is  likely  to  require  key  entry  from 
the  abstracts.    Although  some  errors  may  occur  during  the  keying,  this 
can  be  a  useful  —  although  non-systematic  —  way  of  obtaining  an 
estimate  of  the  type  and  number  of  keypunch  errors  which  may  affect 
data  quality. 

TASK  4.        RUN  TEST  ABSTRACTS  THROUGH  EDIT  PROGRAM 

The  test  abstracts  will  be  edited  once  they  are  entered  into  the 
computer.    The  output  from  the  edit  should  consist  of  an  error  listing, 
a  file  with  "good"  or  error-free  records,  and  a  file  with  the  records 
containing  errors.    Although  each  processor  may  deal  with  erroneous 
records  differently,  some  means  of  flagging  them  as  flawed  is  mandatory 
to  prevent  their  inclusion  in  the  PSRO's  data  base  until  corrected. 
The  error  listing  should  be  returned  to  the  PSRO  for  analysis  with  a 
list  of  the  identifiers  of  each  record  which  "correctly"  made  it  to 
the  "clean"  file.    In  some  cases  a  tape  dump  of  both  gooa  and  bad  files 
will  aid  analysis  by  indicating  whether  keypunch  errors  may  have  caused 
results  other  than  those  expected. 
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TASK  5.        COMPARE  OUTPUT  ERROR  MESSAGES  WITH  EXPECTED  RESULTS 


The  error  listing  should  be  closely  examined  to  verify  the  error 
identified.    This  is,  in  fact,  the  heart  of  the  test. 

In  the  pages  following,  expected  results  from  Run  1  are  presented. 
The  run  sheets  are  labeled  Run  2  for  it  is  in  Run  2  that  the  corrections 
will  be  made.    Each  sample  abstract  is  identified  by  batch  number, 
abstract  number  within  the  batch,  and  patient  identifier  coded  in  the 
sample.    A  number  of  the  test  abstracts  contain  no  errors  and  therefore 
are  expected  to  pass  the  processor's  edit  program.    Each  of  these  must 
also  be  verified.    Those  records  containing  errors  should  be  flagged 
by  the  edit  program  and  some  suitable  explanatory  message  printed  out 
to  indicate  the  reason  for  exclusion.     If  an  edit  program  rejects  a 
record  after  the  first  error  encountered  on  an  abstract,  the  "first 
run"  may  be  much  longer  than  anticipated  because  many  of  the  test 
records  are  coded  with  multiple  errors. 

The  analysis  must  proceed  systematically  to  determine  precisely 
why  discrepancies  exist.    Before  any  edit  program  modifications  are 
suggested,  the  PSRO  must  be  absolutely  certain  that  the  discrepancy 
between  expected  and  actual  results  was  caused  by  edit  program  failure, 
rather  than  by  a  keypunching  error,  or  an  error  in  transferring  the 
data  from  the  test  abstract  to  the  PSRO's  abstract.     In  addition,  if 
locally-collected  items  are  being  tested  with  the  PHDDS  items,  the 
coding  of  these  test  errors  must  be  carefully  examined. 

The  exact  process  to  be  used  in  analyzing  discrepancies  will  vary 
from  PSRO  to  PSRO.     It  is  anticipated  that  the  review  will  involve 
study  of  the  computer-generated  error  list,  the  test  abstracts,  the 
edits  indicated  in  Transmittal  No.  28,  and  possibly  a  dump  of  the 
records  considered  to  contain  errors.    The  sheets  presenting  expected 
results  from  the  first  run  are  organized  to  be  used  by  the  individuals 
checking  the  output  as  worksheets.    Each  abstract  is  listed  with  the 
error,  error  message,  and  correction  procedure.    A  column  for  checking 
off  the  test  errors  is  included. 


V-7 
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VI.     SECOND  RUN 


The  second  run  of  the  test  is  to  make  corrections  to  the  incorrect 
records  entered  into  the  system  during  the  first  run.     After  edit 
processing,  an  error-message  listing  from  the  processor  should  be 
provided.     This  listing  identifies  which  abstract  is  in  error  ana  the 
nature  of  the  error  (s).     The  analysis  of  these  errors  will  identify 
the  source  of  the  error  and  indicate  the  type  of  corrective  action 
necessary.    This  chapter  provides  the  details  necessary  for  making  the 
second  run. 


The  tasks  in  the  second  run  are  the  following: 


TASK 

1 

-  TAKE  CORRECTIVE  ACTION 

TASK 

2 

-  PREPARE  CORRECTIONS  FOR  DATA  ENTRY 

TASK 

3 

-  ENTER  THE  CORRECTIONS 

TASK 

4 

-  RUN  CORRECTED  ABSTRACTS  THROUGH  EDIT  PROGRAM 

TASK 

5 

-  COMPARE  OUTPUT  ERROR  MESSAGES  WITH  EXPECTED  RESULTS. 

TASK  1         TAKE  CORRECTIVE  ACTION 

The  analysis  of  records  on  the  error  listing  should  indicate  the 
reason  the  record  was  in  error.     These  errors,  when  compared  with  the 
expected  results,  should  correspond.     If  it  is  shown  that  the  error 
was  a  data  entry  mistake  (i.e.,  a  keypunch  error),  then  reentry  of  the 
abstract  is  required.     If  the  record  indicates  an  error  in  keeping 
with  the  expected  results,  then  that  record  can  be  further  processed 
in  the  test.    When  discrepancies  between  the  expected  results  and  the 
error  listing  are  completely  resolved,  then  corrective  action,  as 
indicated  on  the  run  sheets,  is  ready  to  be  taken. 

TASK   2.        PREPARE  CORRECTIONS  FOR  DATA  ENTRY 

As  indicated  by  the  run  sheets  following  this  chapter,  changes 
are  to  be  made  to  the  abstracts.     Each  of  these  changes  are  designed 
to  do  one  of  the  following:     correct  the  original  error,  change  from 
one  error  to  another,  continue  with  an  error,  or  create  new  ones  while 
eliminating  or  retaining  old  errors.     Thus  it  is  important  that  the 
recoding  instructions  on  the  run  sheet  be  followed. 

The  correction  abstracts  are  coded  with  minimal  data  or  key  items 
as  indicated  on  the  sample  test  abstracts  in  order  to  allow  the 
corrections  to  be  reassociated  with  the  suspended  abstracts  in  error 
but  already  in  the  system  (unless  the  system  requires  reentry  of  all 
data).  Batch  number,  abstract  number,  patient  identification  and  the 
desired  corrections  will  normally  be  required. 
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If  the  system  involves  on-line  data  entry  with  partial  editing, 
the  PSRO  roust  again  (see  Chapter  V,  Task  1(3))  make  special  arrangements 
to  assure  that  the  results  and  interaction  are  documented  and  that 
all  necessary  information  is  present  for  making  on-line  corrections 
to  the  record. 

In  the  event  the  system  requires  the  erroneous  record  to  be 
deleted  and  a  new  one  substituted,  simply  make  the  error  correction 
on  a  completely  filled-in  abstract  as  if  it  were  -an  original,  first- 
time  submission. 

TASK  3         ENTER  CORRECTIONS 

After  the  correction  abstracts  have  been  prepared,  the  abstracts 
are  the  computer  in  accordance  with  usual  processing  procedures.  The 
same  considerations  described  in  Chapter  V,  Task  3  should  be  reaa. 

TASK   4         RUN  CORRECTED  ABSTRACTS  THROUGH  EDIT  PROGRAM 

This  task  is  identical  to  the  procedure  identified  in  Chapter  V, 
Task  4.     It  should  be  noted  that  the  error  listing  output  from  this 
processing  is  considerably  smaller  than  when  the  test  abstracts  were 
originally  entered.     Similarly,  the  number  of  corrected  records  is 
increased. 

TASK  5         COMPARE  OUTPUT  ERROR  MESSAGES  WITH  EXPECTED  RESULTS 

In  the  following  pages,  expected  results  from  Run  2  are  presented. 
Each  error  record  is  identified  and  both  expected  results  and  further 
corrective  action  indicated.    Most  corrected  test  records  are  expected 
to  pass  the  edits  at  this  point.     Each  record  transaction  must  be 
verified  as  meeting  the  specification  of  the  edit. 
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VII.     THIRD  RUN 


The  purpose  of  the  third  run  of  the  test  is  to  make  the  final 
corrections  to  the  original  test  abstracts  from  Run  1  and  the 
corrective  action  taken  during  Run  2.     After  editing  this  run  it  is 
expected  that  no  further  corrective  action  will  need  to  be  taken. 
Tasks  in  the  third  run  are  as  follows: 


TASK 

1 

-  TAKE  CORRECTIVE  ACTION 

TASK 

2 

-   PREPARE  CORRECTIONS   FOR  DATA  ENTRY 

TASK 

3 

-  ENTER  THE  CORRECTIONS 

TASK 

4 

-  RUN  CORRECTED  ABSTRACTS  THROUGH  EDIT  PROGRAM 

TASK 

5 

-  COMPARE  OUTPUT  WITH  EXPECTED  RESULTS 

TASK  1         TAKE  CORRECTIVE  ACTION 

The  results  of  the  analysis  from  the  previous  run  will  determine 
if  all  the  corrections  submitted  for  entry  were  accepted.  Differences 
between  the  expected  results  and  errors  not  expected  must  be  completely 
resolved  prior  to  making  this  third  run.  When  all  discrepancies  are 
resolved  take  the  corrective  action  as  indicated  in  the  run  sheets 
for  Run  3. 

TASK  2         PREPARE  CORRECTIONS  FOR  DATA  ENTRY 

These  corrections  are  expected  to  be  the  last  corrections 
necessary  for  the  test.    They  should  be  prepared  as  were  those  prepared 
for  the  second  run.    Utilize  the  change  and  checklist  columns  to 
facilitate  the  error  correction  process. 

TASK  3         ENTER  THE  CORRECTIONS 

The  process  here  is  identical  to  Run  2  as  described  in  Chapter 
V,  Task  3. 

TASK  4         RUN  CORRECTED  ABSTRACTS  THROUGH  EDIT  PROGRAM 

This  task  is  identical  to  the  procedure  described  in  Chapter  V, 
Task  4. 

TASK  5         COMPARE  OUTPUT  WITH  EXPECTED  RESULTS 

It  is  expected  that  no  errors  will  result  from  this  run.  The 
number  of  error  free  records,  however,  must  match  the  records  originally 
entered. 
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FINAL  RUN 


VIII.     FINAL  RUN 


It  is  at  this  stage  of  the  test  process  that  a  clean  file  of  test 
data  is  obtained.     It  is  possible  that  the  finish  of  Run  3  will  also 
be  the  Final  Run.    All  known  errors  have  been  successfully  identified 
and  no  more  corrected  abstracts/updates  need  be  generated.  Under 
actual  operating  conditions,  a  PHDDS  tape  would  be  created  and  sent 
to  BQA.     Instead,  the  PSRO  will  request  the  processor  to  "dump"  the 
tape  for  purposes  of  this  test.    The  tape  "dump"  will  then  be  studied 
to  assure  that  no  data  transposition,  loss,  or  alteration  has  occurred 
in  the  process  of  accepting  the  data,  editing  it,  maintaining  the  data 
file,  or  reformatting  or  recoding  the  test  data  from  the  PSRO's  test 
data  base  to  the  tape. 

The  tape  listing  should  be  in  an  easy  to  read  format,  preferably 
each  record  on  its  own  page.  The  PSRO  may  then  examine  each  record 
to  see  if  the  data  listed  matches  the  input.  If  discrepancies  exist, 
the  PSRO  should  identify  the  probable  cause  of  these  problems.  If 
they  are  data  processing  problems,  the  PSRO  should  require  alteration 
to  the  processing  system.  Any  such  changes  must  then  be  tested  until 
all  errors  are  resolved. 

If  the  data  is  determined  to  be  erroneous,  but  not  the 
responsibility  of  the  processor's  system  (i.e.,  wrong  dates  entered, 
but  still  within  legal  limits) ,  the  PSRO  should  consider  whether 
another  test  run  is  necessary  to  clear  up  the  matter.    If  the  problem 
occurred  only  once  and  the  source  of  the  mistake  can  be  readily 
identified  as  a  key  punch  entry  error,  for  example,  it  may  not  be  cost- 
effective  to  request  that  the  correction  be  made  and  another  run 
undertaken. 

Major  discrepancies  may  result  in  modification  of  the  processing 
system;  complete  retesting  of  the  system  should  be  scheduled  after 
the  modifications  have  been  made. 
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IX.     TEST  EVALUATION 


Evaluation  of  the  success  of  the  test  is  an  important  last  step 
in  applying  the  Test  Package.     If  the  processor  has  been  able  to 
generate  a  PHDDS  tape  containing  clean  test  data,  the  PSRO  may  be 
assured  that  the  processor's  edits  are  adequate  to  identify  most  of 
the  common  problems  occurring  within  PHDDS  data.     With  this  result, 
the  test  is  "successful",  regardless  of  the  effort  expended  to  attain 
the  clean  data.     Clearly,  the  processor  whose  system  correctly 
identifies  the  coded  errors  during  the  course  of  the  several  scheduled 
runs  may  be  judged  to  have  more  successfully  completed  the  test  than 
a  processor  whose  system  requires  modification  in  order  to  comply. 
However,  if  the  test  has  provided  PSRO  with  a  satisfactory  outcome, 
both  the  data  processor  and  the  PSRO  may  wish  to  certify  this  outcome 
and  that  the  edit  system  is  acceptable  for  operational  use. 

Among  the  most  important  outcomes  of  the  test  will  be  the 
information  available  to  PSRO  management  regarding  the  performance  of 
the  processor  and  the  logical  correctness  of  the  PHDDS  data  processed 
by  the  system.     The  following  types  of  management  issues  may  be 
addressed  following  evaluation  of  the  test  results: 

Compliance  of  the  processor  with  Transmittal  28   (and  any 
amendments)    and  PSRO-determined  specifications  may  have 
implications  for  subcontract  renewal  with  the  processor 

Responsiveness  of  the  processor  to  identified  system  errors 
and  their  correction  may  have  implications  for  subcontract 
renewal  with  the  processor 

The  identification  of  numerous  errors  in  the  processor's 
system  may  indicate  the  need  for  better  monitoring  and  for 
closer  or  clearer  communication  with  the  processor  regarding 
system  operations 

A  systematic  retest  of  the  processor's  system  may  be 
scheduled  to  occur  at  specified  intervals   (e.g.,  annually) 
as  part  of  the  subcontract  requirements 

Requirements  for  detailed  system  retest  may  be  required 
following  modification  of  the  current  system  or  part  of 
subcontract  requirements 

Plans  for  developing  and  implementing  systematic  tests  for 
accuracy  and  compliance  of  the  processor's  system  in 
generating  PSRO-specif ied  output  reports  from  the  PHDDS  data 
may  be  deemed  useful. 
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In  summary,  the  PHDDS  Test  Package  can  provide  the  PSRO  with  the 
means  for  conducting  an  initial  view  of  the  processor's  system  by 
testing  some  output  from  that  system  to  determine  whether  it  meets 
the  standards  set.     The  results  of   the  test  should  lead  the  PSRO  to 
develop  a  means  for  assuring  that  compliance  in  that  area  is  maintained 
and  to  prepare  for  further  tests  of  other  parts  of  the  data  processing 
system. 
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